Infinispan can be used on the Amazon Web Service (AWS) platform and similar cloud based environment in several ways. As Infinispan uses JGroups as the underlying communication technology, the majority of the configuration work is done JGroups. The default auto discovery won't work on EC2 as multicast is not allowed, but JGroups provides several other discovery protocols so we only have to choose one.
TCPPing, GossipRouter, S3_Ping
The TCPPing approach contains a static list of the IP address of each member of the cluster in the JGroups configuration file. While this works it doesn't really help when cluster nodes are dynamically added to the cluster. See http://community.jboss.org/wiki/JGroupsTCPPING for more information about TCPPing.
Sample TCPPing configuration
<config xmlns="urn:org:jgroups"
xmlns:xsi="[http://www.w3.org/2001/XMLSchema-instance]"
xsi:schemaLocation="urn:org:jgroups file:schema/JGroups-2.8.xsd">
<TCP bind_port="7800" />
<TCPPING timeout="3000"
initial_hosts="$\{jgroups.tcpping.initial_hosts:localhost\[7800\],localhost\[7801\]\}"
port_range="1"
num_initial_members="3"/>
<MERGE2 max_interval="30000" min_interval="10000"/>
<FD_SOCK/>
<FD timeout="10000" max_tries="5" />
<VERIFY_SUSPECT timeout="1500" />
<pbcast.NAKACK
use_mcast_xmit="false" gc_lag="0"
retransmit_timeout="300,600,1200,2400,4800"
discard_delivered_msgs="true"/>
<UNICAST timeout="300,600,1200" />
<pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000" max_bytes="400000"/>
<pbcast.GMS print_local_addr="true" join_timeout="3000" view_bundling="true"/>
<FC max_credits="2000000" min_threshold="0.10"/>
<FRAG2 frag_size="60000" />
<pbcast.STREAMING_STATE_TRANSFER/>
</config>
GossipRouter
Another approach is to have a central server (Gossip, which each node will be configured to contact. This central server will tell each node in the cluster about each other node. More on Gossip Router @ http://www.jboss.org/community/wiki/JGroupsGossipRouter
The address (ip:port) that the Gossip router is listening on can be injected into the JGroups configuration used by Infinispan. To do this pass the gossip routers address as a system property to the JVM e.g. -DGossipRouterAddress="10.10.2.4[12001]" and reference this property in the JGroups configuration that Infinispan is using e.g.
Sample JGroups configuration for Gossip Router
<config>
<TCP bind_port="7800" />
<TCPGOSSIP timeout="3000" initial_hosts="${GossipRouterAddress}" num_initial_members="3" />
.
.
</config>
S3_Ping
Finally you can configure your JGroups instances to use a shared storage to exchange the details of the cluster nodes. S3_ping was added to JGroups in 2.6.12 and 2.8, and allows the Amazon S3 to be used as the shared storage. It is experimental at the moment but offers another method of clustering without a central server. Be sure that you have signed up for Amazon S3 as well as EC2 to use this method.
Sample S3_Ping configuration
<?xml version="1.0" encoding="UTF-8"?><config> <TCP bind_port="7800" /> <S3_PING secret_access_key="replace this with you secret access key" access_key="replace this with your access key" location="replace this with your S3 bucket location" /> <MERGE2 max_interval="30000" min_interval="10000" /> <FD_SOCK /> <FD timeout="10000" max_tries="5" /> <VERIFY_SUSPECT timeout="1500" /> <pbcast.NAKACK use_mcast_xmit="false" gc_lag="0" retransmit_timeout="300,600,1200,2400,4800" discard_delivered_msgs="true" /> <UNICAST timeout="300,600,1200,2400,3600" /> <pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000" max_bytes="400000" /> <VIEW_SYNC avg_send_interval="60000" /> <pbcast.GMS print_local_addr="true" join_timeout="60000" view_bundling="true" /> <FC max_credits="20000000" min_threshold="0.10" /> <FRAG2 frag_size="60000" /> <pbcast.STATE_TRANSFER /> <pbcast.FLUSH timeout="0" /></config>
JDBC_PING
A similar approach to S3_PING, but using a JDBC connection to a shared database. On EC2 that is quite easy using Amazon RDS. See the JDBC_PING Wiki page for details.
Creating a cluster node with distributed cache
Creating a cluster
GlobalConfiguration gc = GlobalConfiguration.getClusteredDefault();
gc.setClusterName("infinispan-test-cluster");
gc.setTransportClass(JGroupsTransport.class.getName());
//Load the jgroups properties
Properties p = newProperties();
p.setProperty("configurationFile","jgroups-config.xml");
gc.setTransportProperties(p);
Configuration c = new Configuration();
//Distributed cache mode
c.setCacheMode(Configuration.CacheMode.DIST_SYNC);
c.setExposeJmxStatistics(true);
// turn functionality which returns the previous value when setting
c.setUnsafeUnreliableReturnValues(true);
//data will be distributed over 3 nodes
c.setNumOwners(3);
c.setL1CacheEnabled(true);
//Allow batching
c.setInvocationBatchingEnabled(true);
c.setL1Lifespan(6000000);
cache_manager = new DefaultCacheManager(gc, c, false);